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ABSTRACT 



Digital libraries are moving from research and development 
into commercial use. If they are to realize their full potential, however, 
the experiences of end users need to be taken into account from the earliest 
stages of design. Those end users are typically individuals who have no 
particular skills in information retrieval, and are accessing library 
resources from their own desks, without support from a librarian. These 
factors clearly have implications for design. The aim of the work reported in 
this paper was to better understand the usability issues raised when digital 
libraries are used in a natural setting. The method used was a protocol 
analysis of users working on a task of their own choosing to retrieve 
documents from publicly available digital libraries. Various classes of 
usability difficulties were found. This paper focuses on "use in 
context " --that is, usability concerns that arise from the fact that libraries 
are accessed in particular ways, under technically and organizationally 
imposed constraints, and that use of any particular resource is 
discretionary. The concepts from an Interaction Framework, which provides 
support for reasoning about patterns of interaction between users and 
systems, are applied to understand interaction issues. (Contains 17 
references . ) ( Author /AEF) 
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ABSTRACT 

The aim of the work reported here was to better understand the 
usability issues raised when digital libraries are used in a 
natural setting. The method used was a protocol analysis of 
users working on a task of their own choosing to retrieve 
documents from publicly available digital libraries. Various 
classes of usability difficulties were found. Here, we focus on 
use in context - that is, usability concerns that arise from the 
fact that libraries are accessed in particular ways, under 
technically and organisationally imposed constraints, and that 
use of any particular resource is discretionary. The concepts 
from an Interaction Framework, which provides support for 
reasoning about patterns of interaction between users and 
systems, are applied to understand interaction issues. 

Keywords 

Digital Libraries, video protocols, interaction modelling, HCI. 

INTRODUCTION 

Digital libraries are moving from research and development 
into commercial use. If they are to realise their full potential, 
however, the experiences of end users need to be taken into 
account from the earliest stages of design. Those end users are 
typically individuals who have no particular skills in 
information retrieval, and are accessing library resources from 
their own desks, without support from a librarian. 

These factors clearly have implications for design. In 
particular, libraries need to be “walk up and use” systems that 
are easily learned; while more experienced users may require 
powerful features that support them in performing focused 
searches, novice users need to get early results for minimal 
effort [5]. The study reported here investigated how 
comparative novices work with existing digital libraries, 
focusing particularly on the patterns of interaction and 
difficulties they experienced. Many of these difficulties are 
not apparent when a library is tested in isolation. Whereas 
users of physical libraries have to move very deliberately from 
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one library to another - often with a substantial time interval 
between the use of one and the use of the next - users of 
digital libraries can move almost seamlessly between them, 
sometimes without even noticing the transition. The focus of 
this study is on use of multiple digital libraries within a 
single session, and on user experiences of the interaction. 

BACKGROUND 

We briefly review related work on use in context and on types 
of use. These other studies provide the academic context for 
the study presented here. We then present an overview of 
methods used in previous studies, relating the methods to the 
kinds of findings those studies could establish. In particular, 
an understanding of the alternatives was used to guide the 
design of the study reported here. 

1.1 Use in context 

Several studies have looked at digital libraries in context - 
that is, not just the library itself, but also how it sits within a 
larger frame of use. One example of such studies is Bishop’s 
[3] consideration of the use of digital libraries by people from 
different social and economic backgrounds. Her studies 
indicate that people from different backgrounds (low-income 
and academic) can easily be put off using digital libraries - 
small problems tend to be magnified until they deter potential 
users, and lack of awareness of library coverage often prevents 
users from understanding what they could get out of the 
libraries. As libraries are becoming increasingly available for 
general use, the finding that people are easily deterred has to 
be taken seriously; our results, as presented below, highlight 
some of the deterrent factors, including poor reliability, 
inadequate feedback and the time taken to familiarise 
themselves with a new library. 

Covi and Kling [7] investigated patterns of use of digital 
libraries by different groups of users, and how they vary across 
academic fields and universities. They focussed on 
interviewing potential users and, moreover, were concerned 
primarily with university members, rather than considering the 
population at large. Their study led them to conclude that the 
development of effective (useful and used) digital libraries 
needs to take account of the important roles played by other 
people within the broader system of use (notably colleagues 
and librarians), and that the views of end users, as well as those, 
of librarians and computer specialists, need to be understood 
for effective design. Whereas Covi and Kling were concerned 
with how library use fitted in with overall working patterns, 
the study reported here is looking at how a particular 
interaction between a user and digital libraries evolves - i.e. 
the patterns of interaction, rather than the patterns of use. As 
shown below, the decisions taken by computer scientists and 
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librarians have substantial impact on the user experience, in 
ways that are hard to anticipate. 

1.2 Types of work with digital libraries 

One approach to understanding users* views is to analyse the 
kinds of work people perform with library resources. In this 
section we summarise studies of the kinds of things people get 
up to, or might get up to, in a digital library. 

Searching for interesting articles is often the first thing that 
comes to mind when considering “digital libraries’*. However, 
as is being increasingly recognised, searching is not just a 
case of entering a search term and viewing a list of results. 
Furnas and Rauch [9] found that in searching for information a 
“one-shot query** is very rare. More typical is an extended and 
iterative search which involves opportunism; that is, the 
searching evolves over a period of time and relies on users 
being able to follow new (interesting) paths as they appear, 
which may not necessarily have been specified at the start of 
their search. 

These notions of extended searching are supported by research 
carried out in conventional libraries [14]. As with digital 
library studies, O’Day & Jeffries found that one-shot searches 
were rare. Rather, they found that single searches evolved into 
other kinds of searching which they identified as: monitoring 
a topic over time; following an “information-gathering plan”; 
and exploring a topic in an undirected way. The findings of the 
current study are consistent with those of others, but illustrate 
that many searches are unsuccessful. 

People do not just search for items in digital libraries, but also 
browse for them. Jones et al. [12] characterise this distinction 
as follows: 

1. Browsing - users traverse information structures to 
identify required information 

2. Searching - users specify terms of interest, and 
information matching those terms is returned by an 
indexing and retrieval system. Users may, in turn, browse 
these results in an iterative manner as discussed above. 

Gutwin et al. [11] discuss the browsing in digital libraries, but 
tend to focus on how user interfaces they develop can support 
browsing, rather than considering what browsing is. However, 
in their discussion of browsing support they do categorise the 
purpose of browsing as follows: 

1. Collection evaluation: What’s in this collection? Is it 
relevant to my objectives? 

2. Subject exploration: How well does this collection cover 
area X? 

3. Query exploration: What kind of queries will succeed in 
area X? How can I access this collection? 

These can all be considered aspects of familiarisation: with the 
type of information in collections and how the library works. 
In the study reported here, many of the purposes associated by 
Gutwin et al. with browsing arise in situations where there is 
no significant “browsing” activity - i.e. no substantial 
traversal of information structures. The purposes of an 
interaction and the types of behaviour (e.g. searching vs. 
browsing) do not appear to be closely linked. Nevertheless, the 
identification of these purposes is helpful: in the discussion 
below, these kinds of purposes are discussed under the 
heading of “familiarisation**. 



Once a document has been located a user typically does some 
work with it (not necessarily immediately afterwards) - 
otherwise, there would be no point in finding it in the first 
place. Amongst the activities associated with working with 
documents are reading and annotating them. One important 
finding about reading activity [1, 2, 13] is that people do not 
simply read articles from beginning to end, but rather move 
between levels of information - for example, from authors and 
titles to reading the conclusion. Current digital libraries 
available via the web are, for various reasons, limited in terms 
of the kinds of reading they support. Even with such technical 
restrictions, there were substantial variations between subjects 
in this study as regards the reading processes they adopted. 

As well as reading, people also create, update, and annotate 
documents. 0*Hara et al. [15] focussed on such writing 
activities in their studies of PhD students’ use of libraries. 
They found that reading and writing were inextricably 
intertwined. Existing web-based library interfaces do not 
support any writing activities, so subjects in this study saved 
and printed relevant articles, for future organisation and 
annotation. 

1.3 Techniques Used in Studies 

Many different techniques have been used to study people’s 
use of digital libraries. This section outlines the range of 
techniques used and their applicability. An understanding of 
these past studies and their scope was used to design the study 
described here. 

Adler et al. [1] and O’Hara et al. [1 5] asked people to keep 
daily notes of their document activities, and followed up these 
descriptions with structured interviews. Such a technique has 
the advantage of low effort on the part of the analysts. 
However, a large load is placed on the participants as they have 
to keep notes of their activities. Moreover, as individuals keep 
their own diaries there will be differences between their note 
taking styles which may be significant and cause problems in 
generalising results. However, such an approach helps us to 
understand what people do and why they do it, which can give 
useful input to design or some qualitative evaluation results. 

Questionnaires, on-line form filling, or registration 
documents can provide simple feedback. Bishop [3] used 
registration documents to build up an understanding of the 
different backgrounds of digital library users. In contrast she 
used surveys to find out information about users* use of the 
system after a period of time. Advantages of using such 
techniques include the ability to get a large response, but the 
information returned is often shallow - typically just simple 
answers to questions asked with no explanations of answer 
rationale. Theng etal.’s work [17] contrasts these approaches 
by using extensive questionnaires with a small group of users 
after they have completed tasks with digital libraries. These 
questionnaires elicit users* perceptions of the digital libraries 
used but, again, they provide no means of assessing why users 
felt as they did. 

Several studies, e.g. [2, 3, 13], have employed transaction logs 
to gain an understanding of the activities users were engaging 
in with digital libraries. These logs give quantitative accounts 
of user actions and so can be used to make statements such as 
“(about 5%) took advantage of the ability to search for terms 
in individual components of articles** [2], However, such logs 
do not provide an understanding of why users use particular 
features of systems. Understanding why things have happened 
is typically tackled by interview and possibly diary studies. 
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Several studies have investigated how people use particular 
interfaces and how their use differs between interfaces and 
between tasks. For example, Bishop [3] and Park [16] used 
experimental design to compare the applicability of user 
interfaces for digital libraries. The use of experimental design 
gives statistically significant results, but such studies are 
costly to develop and run, and can only answer specific 
questions. In these cases, the aim was broadly to inform design 
and redesign of a particular interface. Therefore, the user and a 
single interface could be considered as a “closed” system, 
independent of external influences. This contrasts with the 
present study, for which use in context is the primary concern. 

Bishop [2] used three focus groups to elicit understandings of 
how faculty members used journal articles, and what 
requirements such use placed on design of digital libraries. 
Following on from that, Bishop [3] used focus groups to help 
understand different socio-economic backgrounds of digital 
library users. Although focus groups are useful for gaining an 
overview of the issues and problems, they tend to produce 
information which is often sketchy and in outline form. 

Bishop [2, 3] used interviews to follow up on topics that were 
raised in her focus groups. This approach allows the analysts 
to develop a fuller understanding of the issues raised in the 
focus group, but interviews are time consuming and are, again, 
producing qualitative results which feed into design. Other 
studies such as those conducted by Covi and fCling [7] relied 
solely on interviews to assess people’s perceptions and use of 
digital libraries. Approaches such as those employed by 
Furnas and Rauch [9] involved a combination of techniques as 
they used interviews to inform further observation of people 
using digital libraries. In contrast Marshall et al. [13] used 
interviews to follow up other techniques such as examination 
of transaction logs; their interviews allowed analysts to probe 
why users were performing certain patterns of interaction 
identified from the logs. 

Observing what people do as they use systems is a time 
consuming activity. However, it can provide useful insights 
into the usability of systems. Bishop [3] discusses her use of 
observation to gather information on engineering work and 
learning activities. This information can then be used to 
inform (re)design, and/ or followed up in other ways such as 
interviews, as illustrated by Furnas and Rauch [9]. 

From the view of the studies presented here it is clear that 
although there has been work on studying the usability of 
specific user interfaces for searching, and to a lesser extent 
browsing, in digital libraries there has been little work on 
understanding the nature of these tasks or how libraries are 
used in a natural setting. 

METHOD 

The study reported here aimed to achieve a better 
understanding of how users interact with digital libraries 
within a single session, but not necessarily using a single 
library. 

Because we wished to gather detailed interaction data, 
techniques such as diary-keeping, interviews, transaction logs 
and focus groups were inappropriate. A video-based 
observational study with think-aloud commentary was 
selected as the most appropriate means of gathering data, with 
a short debriefing interview a few days later to clarify any 
issues raised by the video data. 



Five users were recruited for the study. Three of these were first 
year PhD students (referred to below as “A”, “B” and “C”), one 
a final-year PhD student (“D”) and one an experienced 
academic (“E”), all computer scientists. A - D were recruited as 
subjects specifically for this study; E, aware that this study 
was being conducted, offered to participate while performing a 
self-defined library searching task. 

The aim was not to give users artificial tasks, which are liable 
to be either too precisely defined to be natural or too 
meaningless for participants, but to ask participants to select 
their own tasks to work on. Therefore the task defined for 
participants A - D was simply to obtain at least one paper on 
their own research topic to help with their literature review, 
using their choice of libraries from a given set (easily accessed 
via bookmarks in a web browser). They were asked to think 
aloud while working. They were provided with a little 
information about each library, as shown in Table 1. Access 
rights for each library were defined by the subscription held 
by the organisation in which they are based. Note, in 
particular, the restriction on ACM access - a cause of 
difficulties as discussed below. 



Table 1: bookmarked libraries for users A - D. 



ACM Digital library 
www.acm.org/dl/ 


Full text access only to 
journals and magazines (not 
conference proceedings) 


IDEAL 

www.idealibrary.com 


Access only to articles prior 
to 1998 


NZDL 

www.nzdl.org 


Full text articles 


EBSCO 

www-uk.ebsco.com 


Full text articles 


Emerald 

www.emerald-library.com 


Full text articles 


Ingenta 

www.ingenta.com 


Full text articles 



As noted above, E was not recruited in the same way, but 
offered to participate. She was planning to search for articles 
on particular topics to help with writing academic papers, and 
volunteered to do this with a video camera running, and to 
“think aloud” while working on her self-defined task. 
Consequently, she used digital libraries of her own choosing, 
and did not have explicit information about limitations on 
access. She held a personal subscription to ACM, so she was 
not subject to the same restrictions as other users. 

Users A, B, C, D and E worked with the digital libraries for 57, 
62, 62, 51 and 80 minutes respectively. The video data was 
then transcribed, including speech and some description of 
interaction between user and computer system. It was analysed 
using the Interaction Framework [4]. Extracts from these 
transcripts are used in the following sections as source 
materials for examples. 

1.4 Interaction Framework: overview 

The Interaction Framework is an approach to describing actual 
or possible interactions between agents (users and computer 
systems) in terms of the communicative events that take place 
between those agents, and the patterns of interaction. Using 
this neutral language, which aims to take neither a user- nor a 



computer-centred view of the interactive system, we can 
identify and discuss properties of the interaction that might or 
might not be desirable. For example: 

♦ Blind alleys are interactions such that the objective is 
unachievable, but where that fact does not become 
apparent until some way into the interaction. 

♦ Discriminable events are ones that another agent can 
easily choose between. 

♦ A canonical interaction is one that achieves all its 
objectives as efficiently as is theoretically possible. In 
the case of digital libraries, such objectives will include 
accessing particular papers, accessing papers on a 
particular topic or general subject area, and gaining 
familiarisation with a collection content, type or 
features. 

Central to any successful interaction is the idea that users 
must have an adequate understanding of the state of the 
system. Put in neutral terms, the user and system must share 
“common ground” [6] - that is, each interactional event has to 
communicate sufficient information to enable the agents to 
maintain common ground. 

SUMMARIES OF INTERACTIONS 

Before presenting detailed results, summarising the important 
difficulties found, we present a brief overview of the 
interaction of each participant with the available digital 
libraries. 

1.5 User A 

User A was interested in papers on electronic commerce. As she 
started working, she spent a while browsing unrelated material, 
as if orienting herself to browsing, before selecting a link from 
the bookmark list. She selected the ACM digital library, and 
spent some time reading through the introductory page. She 
then searched for “the best of electronic commerce”, but 
seemed rather confused by the results returned. She selected an 
alternative search mechanism and repeated the search. She 
found various articles that were “interesting” but did not print 
them. Although she limited her search in ACM to “journals 
only”, conference articles were listed among the search results; 
when she tried to download one of these articles, she was asked 
to enter a user name and password, which she did not have, 
resulting in an authorisation failure. 

She moved to the Computer Science Technical Reports link in 
NZDL, and got over a thousand hits on her search. She 
reformulated her search several times, and still too many items 
were returned. Eventually, she found an article she wished to 
view and download, but failed to download it, for reasons 
discussed in section 5.1 below. She then moved on to EBSCO. 
When her search results were returned, she commented that 
“this is more readable than from other libraries”. She 
successfully found, saved and printed one article. 

1.6 User B 

User B was looking for material on knowledge management, 
text mining and link analysis. Although he has used other 
browsers, he did not have previous experience of using 
Netscape, so he starting by browsing Netscape pages before 
connecting to the ACM digital library via the bookmarks. 
Before specifying the search terms, he spent some time “trying 
to understand how to make the search”. When searching the 



ACM library, he did not restrict his search to journals only, 
and consequently received many hits that were conference 
proceedings; like user A, he got authorisation failure when he 
tried to print any of these. 

Although new to Netscape, user B was a relatively 
sophisticated user of information retrieval systems. For 
example, he understood how to use quotation marks 
selectively in search queries and also resorted quickly to 
using two browser windows so that he could continue working 
in one window while a document was downloading in the 
other. Using this strategy, he explored the EBSCO library, then 
NZDL, then Ingenta - continually flicking from one window to 
the other as pages were loading. One apparent consequence of 
this is that his behaviour was more reactive than that of the 
other users: he appeared not to form clear beliefs about the 
state of the system, but to simply respond to whatever was 
currently displayed, and the interaction appeared relatively 
unstructured and haphazard. 

1.7 UserC 

User C was searching for material on Growing Cell Structures 
(GCS), text classification and Self Organising Maps (SOM). He 
started by accessing Emerald, but rapidly switched to Ingenta, 
when his first few query formulations returned no results. 
Results from Ingenta were more promising; in fact, there were 
so many hits he appeared to be overwhelmed: “I found over 
three hundred documents here”. Several times, he selected an 
article with a promising title and followed the link “full text at 
Science Direct”, but was then refused access. [The user 
organisation had a subscription to Ingenta journals but not to 
Science Direct.] He saved several abstracts and printed one full 
text article from Ingenta. He accessed the IDEAL and ACM 
libraries twice each, but each time judged them “too slow”. In 
EBSCO, he failed to find any matches to his search terms. He 
returned to the Emerald library, and reformulated his query: “I 
think I made some mistakes last time. I searched for GCS as the 
keyword so this time I searched for GCS for full text and I 
found something”. Later on, his search took him to a link 
“order the book”, which took him from the library to an 
internet bookseller. He conducted further searches, using 
Emerald, Ingenta and NZDL, usually receiving either no 
matches or too many to deal with. In NZDL, he found an 
interesting article, but failed to download it. Although he 
could read the article on screen, he tried three times to 
download it, without success. By the end of his interaction, he 
had found some relevant material on text classification, but 
none on GCS or SOM. 

1.8 User D 

User D looked for different things in different libraries. He 
started with ACM, searching for articles on usability 
evaluation; a large number of results were returned. He tried to 
save a selected article to a “binder”, but received 
“authorisation failure”. He moved on to use NZDL, now 
searching for articles on “musical timbre”; although he found 
one that he skim-read on screen, he moved on quickly to 
search for “timbre perception”, which returned results that he 
judged “more or less similar to my previous query”. He opened 
second and third windows to access EBSCO and Emerald, but 
never switched between windows (they seemed just to be a 
historical record of where he had got to in a particular library). 
He browsed and submitted search queries in a relatively 
unstructured way, apparently trying to familiarise himself with 
the content and structure of the library. When he tried to 
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follow links to particular journals, an error message, “Type 
mismatch” was displayed. Subsequent tests indicate that this 
was a temporary error, but it had a strong influence on this 
particular interaction. 

He tried to access Ingenta, but received an error message: 
Ingenta was unavailable. He moved on to Emerald, about which 
he commented: “It’s actually good to have some basic 
description of the journals. It’s not my area.” After a quick 
look, he returned to ACM to search for articles on visual 
texture; on selecting a “Find related articles” link, he found 
something quite different that interested him - a case of 
serendipity in the search. He viewed the abstract and tried to 
download the paper, but got an “authorisation failure” - he 
“wasn’t paying attention” when earlier told that he could 
download journal papers but not conference proceedings. He 
went on to successfully identify and download a relevant 
journal article. 

1.9 UserE 

User E required information on diary systems, cognitive 
modelling and usability of Artificial Intelligence systems. In 
meeting these objectives she used three libraries: ACM, IDEAL 
and the New Zealand Digital Library (NZDL). These were used 
in a relatively orderly sequence of ACM, followed by IDEAL, 
and finally the NZDL. The search objectives were repeated to 
some extent with each information source as opposed to 
meeting each objective in sequence. 

In detail, she spent most of her time using the ACM DL. Any 
articles she found that seemed relevant were printed for further 
review. Her first few searches were unsuccessful, yielding “no 
matches”. A search for information on diaries and calendars 
was more fruitful. She then switched to browsing various 
collections, with mixed success. When she found an article 
that appeared interesting, she always printed it out, rather than 
trying to read it on screen. Overall, this user’s interaction was 
the most fruitful, and by the end she had printed about a dozen 
articles. 

RESULTS 

Clearly, each of the five users in this study behaved in quite 
different ways when navigating the libraries, and the sample 
size is far too small to make any general claims about usage 
patterns or typical behaviours; this was not the purpose of the 
study. Similarly, the purpose has not been to conduct 
usability evaluations of particular libraries, or to pit them 
against each other. Rather, the aim has been to identify core 
usability issues that arise through the details of user 
interaction with digital libraries, where the users are not 
assumed to be interaction retrieval experts, and where use of 
any particular library is discretionary. 

We discuss the main usability issues raised within this study 
under two headings: deterrent factors (and encouraging ones) 
that may influence future acceptance of digital libraries; and 
usability issues that arise as a direct consequence of libraries 
not being “closed” - i.e. that the interface between a library 
and other resources is easily traversed. 

1.10 Deterrent factors for use in context 

The users in this study all experienced substantial difficulties 
with using one or more of the libraries they accessed. Many of 
these difficulties resulted in blind alley interactions - that is, 
interactions that did not achieve the user’s objectives. Some 
of these blind alley interactions took place over several 



minutes. For example, user A spent over six minutes trying to 
download an article from one library before giving up, 
commenting that “I’ve tried all, but I can’t download”. Some of 
her sources of difficulty are illustrated in Figure 1, where the 
warning message reads: “Expanding the text here will generate 
a large amount of data for your browser to display”. 
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Figure 1: difficulties with downloading in NZDL 



Most users suffered from a “triumph of hope over experience”, 
repeating the same unsuccessful action several times without 
modification, and each time receiving the same result. For 
example, user B tried eight times to download full text of an 
article from Science Direct, while user C did the same six times. 
User C also tried to download the same postscript file three 
times, each time receiving the error message “Netscape is 
unable to find the file or directory named 
/pub/techreports/1 993/tr-93-025psZ. Click the file name and 
try again”. Similarly, user D tried three times to access the 
Ingenta library, each time receiving the error message 
“TfeLogin: Data decoding / encoding error”. It is too early to 
follow up these users to find out whether or not their 
experience working with these libraries on this occasion has 
deterred them from any future use, but the productivity of all 
participants in the study was low (one document successfully 
retrieved after an hour of use for all except user E). Even the 
most successful user, E, commented at the end of the 
interaction that: 



„.I don’i jhinkjijcb cope with .doing anything more.: I don’t'' 

Teel that I’ve reallyifbund much^apartTrom on calendars j which 
I " c quid focu s. I ! hi ay e n ’ t ; f ou nd ve ry much; tti at was ^ 

'helpfuL . ' ."iiy.'isi-.*" • • V-i "i'-' 



The downloading difficulty of user A can be understood in 
terms of poor “common ground” between user and system - 
that is: the user did not fully understand the state of the 
system, or the meaning of the warning message, and 
consequently could not find a way around her difficulty. 

The errors experienced by users C and D can be understood in 
similar terms: that the event communicating the fault to the 
user is insufficiently expressive for the user to comprehend 
and respond appropriately (see Figure 2). Accepting that 
occasional errors such as network faults are unavoidable, the 
challenge to the digital library designer is to communicate the 
status of the system (which may include multiple servers) 
effectively to the user. 
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Figure 2: users did not understand the difference between the 
two sources of full text 



Gould [10] discusses the importance of reliability when 
considering usability: a computer system that cannot be relied 
on to work predictably well is difficult to learn and use. 
Digital libraries accessed using standard web technologies are 
complex systems that are vulnerable to many kinds of failure. 
Some of the failures affecting basic accessibility of the library 
resources have already been discussed. Others have a more 
subtle effect on the interaction. For example, user A, aware that 
she could only access full text of journal papers (not 
conference proceedings) from ACM, conducted a search 
restricted to journals only. The following transcript shows the 
user’s words, her [>actions] and the [<system feedback]: 

. so 1 will search in ‘‘all j ourn ail s ;and,pr oceedi ngs’!$: no , 
‘journals only’ -i*. 1 •' ^f : 4r • “• : ; 

![>. selects: ‘^purrialsiicmiy’^ i fr; : y; / -fe f ii.!; 

[> clickslpn : ‘‘ search’’ button]' •' . ; '' ■■ *. • v.' 

' [< ; : AOftf -page] 

. A I -found^ ; .• . 

interesting.:^ .markets‘ u arid dntelligent;systems ! . 

This is|interesting f •: 

■! 1 !t ? ; Ki; ; *, : \ ;V Jr: : 

. . . Building bridges ? ,with .practice; . :.T -wlljhave a look at 

this..; '=• v . !* jT| v : 

[> clicks' pnithe document link ] . / t ^ : : * • - V yJtf: 

[< article; - abstract page appears] • A \ . . : ” A 

. . . thi s is very close to what' I am interested in : 

[> scrolls' (dbwn 'tlie page] A y%'j, / ; {'•* ,j ... , ! -"l; : . 

[< bo ttom ;o f abstract . artic le page :di splayed];! ■ . • V. v : . --• 

[>. clicks, on 

[< the pop-up window “User name and password’’ appears] 
Ow yaa. You have to get a username and the password: 
because} jjiayeii \ t registered ". : -• /-W 

Her confusion lasted some time longer. What appears to have 
happened here is that there was a temporary bug in the ACM 
library, or in the means by which the user’s search 
specification was transmitted to the search engine, such that 
the search was not restricted as specified. Consequently, many 
of the results returned were articles from conference 
proceedings, but the user did not check this, believing that she 
had restricted her search to eliminate such results. A few days 
later, when we tried to reproduce the interaction as recorded on 



videotape, the problem had been corrected. Accepting that 
such errors are not always avoidable, it should be possible, as 
discussed above, to give more focused feedback, so that the 
user can understand and respond to the error in an informed 
way. In this particular case, the user was restricting her search 
because she was aware that she had authorisation (based on her 
host organisation’s subscription) to access full text of journal 
articles, but not conference papers. Arguably, the system also 
has access to this information, and might have been designed 
so that the user’s access rights were clearly indicated before 
she tried to download an article. 

As well as failures, the user’s choice of libraries is clearly 
affected by the quality of their experience with a library, and 
also their sense of familiarity with that library. Within the 
study reported here, the three dominant factors that determined 
perceived quality were: a sense of making progress (whether in 
finding relevant articles or in understanding a library better); a 
sense of the task being manageable (in particular, not an 
overwhelming number of alternatives with poor 
discriminability); and system response time being acceptable. 
We consider each of these aspects in turn. 



1.10.1 A sense of progress 

All participants had the experience of issuing search requests 
that returned no matches. Particularly for user C, this became 
very frustrating. E.g.: 

.1 searched this authoran several digital libraries and ’• 
always cannot , get anything ...;this profess o r has pub 1 i shed 
several papers" in! computer journals. - : \ 

And: 



This haven ’.ffourid 'anything about ' * ‘ (j ( i: S ’ f;t do jl. sK* n! ^ 
Where users have a sense that they are learning from their null 
results, they still seem to have some sense of progress. E.g. 
user A felt able to modify her search: 



iUV 



•[> clicks on the ‘‘search’’*button]ii: 

[< “search result” >page appears] 1 , . ;jf ‘.V * v ; * 

A .no ;m^ches^^ found ^ Perhaps 4 If ; 

■sjhou ldn’t pufir ans ££$ public at ion s . ^ j :i v ; 




•journals,' . 

Similarly, user E modified her search, but was still 
unsuccessful: 



[< search Ire suits, pageYr t eplaces search-formu lation page] J; ; 
No matches:' Great; ‘ • 'f:4: v 

[>, clicks back]; /; • , ' [/Jy ; i $ • ’ 1: ' 

[< search 1 page\replaces|search results page]'.::. ;- ' • ' 

So even on fhllitextifGf course^its possible;; lio^r.;.', ■ : : t \ 

;What-happehs,iW the human^andtry again^rvi^. '■ i 

[> removes human from subject search terms] : , V- 
([<: subject I s^rchBsj^wiI^^ 

author searches stiii : “Hollnagel”;)] v- K? • ' k* : 

1 would .expedjfoj'get ( a ,! fair:,amdunt’.. • ;• 

[^clicks searcHjv ». .'-V* 

; [< search; results -replace search formulation page] . . I ' • 
.No; matches^ , . A jy . ' : . ' ' v; ( 'fi. ’ .. ■ 1 .« ^ • • •’ '%£- !, • . 

In this case, the source of the problem was that this search was 
conducted some time after an earlier one that has specified a 
particular author, but the user had not noticed that there was an 
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author entry in the search specification. From the user’s 
perspective this was a new context, a new query; from the 
system’s perspective, this was an elaborate query (with no 
matches). Improved feedback - helping the user to understand 
exactly what the search results referred to - might well ease 
such situations and improve the user’s familiarity with the 
system. 



Conversely, users appeared to have a particularly strong sense 
of making progress when serendipity worked for them - that 
is, when they came across an interesting item that was 
unexpected in the current context. This happened twice to user 
E; e.g.: 



'[- ACM DL list of papers in most recent ; blS?conference] : 
Triangulation.; * ' ' ’ • ; r 

That actually looks quite interesting. ;i 

[> clicks on paper’s link] 

Sb’ I ’ 1 l;ha^ ,ione|i^{ ; iv 

*[>, clicks acrobat print' button]- " , ^ 

It also happened to user D: 



FmTookingdntp a visual 'texture relat^ ; papers,buf h 
actuall^found.something. that doesn’t;seem:to be related to 
that at all: It’s about comparing two different methods^jof-V ' 
evaluations: empirical testing and walkthrough methods ' 
wh i ch i s; a 1 so interesting do me so: . . 1 * 11. j ust j u nip; to th at 
ifrom the search . • 

While serendipity is difficult to design for (by definition), it 
can be supported through discriminability: it is important that 
it is obvious to a user when such items come into view - that 
the descriptions of items make their nature clear. 



1.10,2 A sense of the task being manageable 
While no matches is clearly not a desirable result, there has 
been little discussion of the effects of too many results. 
Within the Information Retrieval community, one focus of 
research, and one criterion by which the quality of a search 
engine is assessed, is the quality of search results returned, as 
measured against suitable metrics. In practice, the users in our 
study appeared to place more store by quantity, or at least 
discriminability, than quality. For example we have the 
following comments from users in response to various search 
results: 



User A (having specified “some” of the search terms to be used 
in a search for “electronic commerce”): 

Here ! found., oiie thousand iseyen hundred 

commerce , andd wo thousand® j ' • 'y! [ .> ■'% T' si '* 

. . iOR^wi^ '-F : \ 

[>' c 1 i ck s f on|;‘^ ack ’ ’ button];: A ," a • U“v 

User C commented: 



Sometimes the search finds too many articles so it is a ; > 
/little -bit boring; ttyreadiall these article' titles.- : . ' •* .. ' * ■ .? 

More encouragingly, user E was pleased when fewer results 
were returned: 

4S|dwuments^ thatno.t Too bad: r’ ■' 

As was user B: 






‘;.U6hly ise;^ • -A' 

Repeatedly, users commented positively when the number of 
results was small (less than twenty) and negatively when it was 
large (typically over two hundred). We can understand this in 



terms of the discriminability of events and the number of 
alternatives: with a large number of results, it is generally not 
possible to discriminate between those possible fhture 
interactions that are likely to be successful and those that are 
blind alleys (relative to the user’s objectives). 



1.10.3 Response time: the pace of the interaction 
Due to factors such as network bandwidth limitations, the 
geographical locations of servers, network loading, etc., the 
response times of libraries were very variable in this study. 
Dix [8] discusses the importance of the pace of the interaction 
being appropriate to the task properties. Two of the users in 
this study (B and E) responded to the pace being too slow by 
opening a second browser window and interleaving interaction 
with the two windows. (D also opened multiple windows, but 
only used them sequentially.) Both B and E commented on 
download time. For example, user E noted that 

jiWe’re* going to work much; more; efficiently if we have :a 
;new\naATj^ator'* window 

[[^clicks, browser ‘file^diew navigator window] Aj 

[<:New, : wihdow (2$*windqw) appearsfwith hdme page^i^Sb'l 
i.^and have jthaf'qhe jninnii^ 

As switching between windows, B commented that 



^fiys^king quite aj>16^ [M] ..•• ; spT,.cah;:;;.' 

continue working t wi.tfr;ApM ^ 

C also reacted to the response time of certain servers, but 
responded in a different way - by avoiding those libraries 
completely: 



[> clicks on IDEAL link] r f 







!"[,< iisf.pfjinks d i sp> 1 ay ed ^ I;;*! 
:t^sele.cfs^jCM DL frqm'tlie bookmarks] . , 



While the cognitive demands of multitasking in this 
environment are not excessive (because neither task is time- 
critical), the similarity of the two tasks is liable to cause 
interference. Both B and E appeared to lose track of the state of 
the second window at times. 



1.10.4 Familiarity 

As noted above, user E made her own selection of libraries; for 
the other four users, information was intentionally re-ordered, 
both on the bookmarks list and on the accompanying paper 
documentation, so that users who followed the list order of 
libraries would naturally start with different libraries. 
Nevertheless, four of the five users started by working with the 
ACM library. Two (D and E) stated explicitly that this was 
because they were familiar with that library. For example, D 
commented: 

:..I*ye usedfhisTibrary in the past so more or less I: know 
[that Ixanffind -relevant -papers?, ;*.£what I'm- looking for.- 
B commented about a library that he had not used before: 

i!;:.lef s tryitHiSone; EBSCO on line;:T don’t know itbut it’s 
always good to know new -things; 0 ; yf% . 

r.;whatsliall l do; v;: I am frying To find out how should : I 
^perateThisV. . Tayorite;*j punials,;let • s-try it. . • \ t '.‘'M* 
:[-^ciiclcs. on “favorite 
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[^‘Manage Favorites” pop-up window .'appears;6n'tHe top ;- « 

l|vj^am trying 'to understaitf ^^ w 

These two examples illustrate familiarity (or lack of it) with 
different aspects of library use. D is concerned with the type of 
content (and his expectation of being able to locate relevant 
papers), while B is concerned with how to work with the library 
system and the navigation features it offers. 



Other aspects include familiarity with content of collections or 
articles - type (e.g. the kinds of papers that are published in a 
particular journal), structure and detailed content. For each 
aspect of familiarity, the design challenge is to support 
recognition (e.g. E, reading the title of a paper: “I happen to 
know what that work is”) and incremental learning. For 
example, user E had difficulty understanding the “binders” 
feature in the ACM library: 




J’mrmeant tado ; wi tliyti erir. ;Hmnr. • I jflon ’ t knbVjwhat'a^,^ ;! 

jbi n de r; is; ■■ V t .7;. f • ‘ t’ {'£ 

;[>. clicks and holds pnjjnn^ : 1 ] ;;V' > 



;[< browser^ menu of possible actions]' . 7 V‘-r 

[> releases mouse] . ‘I \ P 77 vv' 

[< page is replaced with pageicontaining list'd f articles "• 

•selected] ' 7 1 ' '* v 7 • • \ .. ( f r , ' * . v y. , . • > 0 * 

‘..ypleasei, choose a ; binder!;fro^ ’ ,• ’ .... 7 .7’ 

I don’t have a •binder; and I donlf haye, a bookshelf as far as 
1 ■ m aware . . ' : \ : .i ! ’• )\ ji;;. )cyl ’ •; . •, 1 ' 



One of the substantive challenges in designing usable, useful 
digital library systems is enabling users to learn features 
incrementally, so that they are not overwhelmed at the outset 
by a large number of alternative actions that are 
indiscriminate (because the user has no way of predicting the 
effects of any actions). This would include distinguishing 
between core and discretionary features (e.g. document 
management features can be learned after document retrieval 
features), and presenting information in context - at the time it 
is needed - as well as making novel features self-explanatory 
as far as possible. 



1.11 Using multiple libraries 

The main purpose of this study has been to focus on use in 
context. Many aspects of context have been discussed above 
while considering use of individual libraries and deterrent 
factors. In this section we consider additional difficulties that 
arise because libraries are accessed by users from a particular 
organisation, using particular technologies that interact in 
ways that may not have been anticipated. 

Users were all working within a University setting, using a PC 
running the Netscape browser to access library resources. The 
setting and the task were designed to be as natural as possible. 
Two main classes of difficulty emerged in the sessions: 
understanding the access rights they had based on 
organisational subscriptions and working across boundaries 
(between libraries, other software, other systems). We consider 
each of these aspects separately. 



7.77.7 Access rights 

As described above, user A had difficulty downloading papers 
from ACM because she believed she had limited her search to 
those items to which she had access rights, but others were 
listed in the search results. User B also tried to download 
conference papers, having ignored the details on the user 
instructions. Similarly, as described above, users had 
difficulty understanding the difference between papers that 
could be downloaded via Ingenta, and papers that could not be 
downloaded because they were only available via Science 
Direct. A further potential difficulty of the same kind resides 
in the IDEAL library, where users have access rights only to 
journals from one publisher, but not others. These distinctions 
are poorly understood by users and inadequately explained 
within the libraries. Also, in the current dynamic environment 
where new resources are coming on-stream rapidly and 
spending decisions (e.g. on subscriptions) need to be reviewed 
frequently, users have great difficulty keeping track of what 
resources are available to them. Particularly as users are often 
working in various locations, without easy access to the 
information about permissions settings, that information 
needs to be made easily accessible at the point of need. Of the 
five libraries used in this study, one (IDEAL) has recently 
added a feature to give any user individualised information on 
their access rights, which is clearly recognition of the 
difficulty and a first step towards addressing this particular 
problem, which is just one aspect of the broader challenge of 
enabling user and library system to establish common ground. 



7.77.2 Working across boundaries 
Within this study, there were many cases of users traversing 
boundaries - between collections such as Ingenta and Science 
Direct, as discussed above, or between NZDL and various ftp 
sites, as exemplified in user A’s interaction: 




; ilet * s'try; the ’ fi rs t one • 
i![>; clicks on the, first’ 1 i nkl * 77 ' 

re •; •• '1 -V. . V. V 

; ;[<ftp :// actpr.cs. vt:edu page appears] . 0.1 > 
vlsoj ianjt find easjly w]^ .lppiihg‘;’fo;r: • • 



Such transitions, which are often not well marked for the user, 
demand that the user switch to a new way of working, with 
different navigation and other features, in an environment that 
looks and behaves suddenly differently. While transitions 
may be necessary, even desirable in some circumstances, their 
implications for users need to be better understood. 



Because libraries are commonly made available via the world 
wide web, there are many cases where users can accidentally 
leave the library, following links to other web based resources. 
User C experienced this when following a link from Emerald 
that referred to a book: further information about the book 
could only be accessed via internet book stores: 

. J^:pressesJ“Review” button] . : ; 

[< “revi ewiarti cleVpagedispl ayed] \ ; 

i^scrolis^idown^thc'-jp&gc] 

[< ( ’ • }>■ ,/:.*!• : 
[^s^nsjp.the page]’ 7 } ( • f ' 7 •,/;V7 • 7 • ' • 

• ‘ :, v v,. . / ; 7 777 «. ' V • 

4?m;;;t^ • • -' 7 . ’ . % . ) t ‘ : ' 

scroll s \ do w n ■ the! page] : " f ' ' •' “ ; ‘ , . 7 ft.- : Z ’ ! ' ' • . •• 

.[<Sb1tpm::p^ displayed] ; 1 ' ’ - 
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! [>j clicks ori^order the^book’’] [ amazon] ;Vi> j 
j [< • -amazon” page 'displayed] si •' : ?:f / * . • : i 

User A also accidentally left a library - twice - as she searched 
for library resources. The first time was almost immediately 
after she had followed the ACM link from the bookmarks 
provided: 

?:what’s riew^MaybeiifjtW therijit wilfbety-' ’! 

,more>convenient forjme. . N vvf 

; [^clicks on the “search acm” button] 3S? 

; [<“ ACM s earch p age, displayed] ^;jj | . ; .«••. • ^ j\ >i;j 

..:are not included. What is notjincluded? * ^ 

[> typesMhe best of electroniScommerce^info search' Ibox] 
J:it =j^ ••Yes: 1 • ;'?^W M-; '•> 

«[<Ykey worcTpre- selected]/ ! ?' ;l!.' i ’■'-■•’j' rV/r’! • 
i[> l -cH|ics;;ph “ search”; button] ?!, .. • ' // .;;*•>.? ^ vi’ % 

[< “Search -Result” page' appear^ 

•I 1 have 83. -pages matching my query and T have here 1 ; to 12^ 

' '' L ri - - •’ - ,L - ■ ■ 1J - x - - L " -- 1 - lj search! 




re*** ** 'gMs wiywt 
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larieiim tfc«>-CM tL-oitfi luvry, »i 
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The links on the top 



button bar take the user 
out of the library. 



U W» Tttt V* rmt r. r *i., oi-*J U 

ft! ty*Lr» «T r\ tv 

o*y*t4ir*v«r« *r itMurciMfl 

,1 1>4> ‘ s> • •», vftfj .y<; 

Cv? u>> tj&nant ri h 3* « 

*$***:: ri ,-d **«&«, «■ 

*®d) « v* id/crcsa Msw-yiard 
&*a£*t 4 ; t» 1«y 



Figure 3: identifying the exit routes from a library 

She is unaware of the fact that she is not using the digital 
library, but searching the ACM web site. This is the behaviour 
of a novice user who has just entered the ACM digital library 
for the first time, but she has clearly failed to discriminate 
between “search ACM” and “search the Digital Library”. This is 
partly an artifact of the study, since she entered the library 
directly from “outside” without passing through the ACM web 
site, but such situations are common, particularly with the 
increasing provision of digital library portals that provide 
links to various libraries from one web site. 

Later in the interaction, she again left the library - rather more 
briefly - as she tried to return to the digital library home page: 

:[> AGM DL search page displayed] > : ’ 

[>: click on “home” button] ' • 'v-: * . y'.r -L’. 

,[< ACM - home page] . . . ; ? ' : / . ; v 

So, Lwould like^t6;ffiiid..somQ journals; farticies^ ;t; 
magazines... ' ; -f.; 

... So where can I find them. . 



j [>:; clicks on, . V di g i ta L 1 i bra ry^l- 1 i nk] ff^iv : '/V ; fjg% ;: ; ! 1 

Figure 3 illustrates the source of these interactional detours: 
the user has failed to discriminate between the type of links on 
the top links bar (which are generic ACM links) and those on 
the body of the page, which are digital library links. 



As well as boundaries with other web based resources, users 
also have to work across boundaries with other software and 
the operating system. For example, user C had some difficulty 
saving a file, shifting attention from library use to working 
with the operating system. User B accidentally closed his 
browser session, and had to start again: 

) [<-iprtSef pages displayed]'? £ ; -f . .-.S?! ■ ■: V !; 

..; t npf‘in3eresting;\‘|^g-:. 

,[> closes:, the ^etscape^ windoiiy] w ; ::•> : i : 

|[^^pr^document page displayed] : !’?. ’1 v 

1 [> closes: the M i cr osoft*! w 6 rd-’^appl i ca't i o ri] / ?' v ' • 

[< pop-up .window ‘. f Micr os oft -Wo*rd? f appears “Do you want • 
fto save changes in, the' do cume nt”].4’*|v^; '! '?$•' v-’W: •' ?' 

! [^|C^ks ! “np” bu tt 0 n] jy J' • . !:v;; 5> ' 

iftl got out of ( the “Netscape”-!/,? ?'t /•’ ?'/; :\ 

[>plicks‘ on -“-Netscape” icon] :;.!?’ }}$. ;f 

[< “Netscape”; window' opens] ; • 4 ’ . ! y r " 

: .bookmark s:/; now, aga in i . It S: i ?:!; L •'>: 

As the only user in this study who printed files, rather than 
saving them, user E also had to integrate library work with 
using other resources - notably a printer: 

It’s all tied-, up., for the, minute, ^ so f I’ll /go J and cqllectispme r ;?!? 
’thmgsTrom^ liter/ 1^!; i ^ ; r - 

' (| - [| ACM - DL, 1 ^window showin g o ntology; ; p aper] , y , | 0 
J C) S o f I ; ? do n lmojw;: h o w.; I ^ w i 11 laiowifi^ 

• I *ve printed 0 u t? Becaii se I ’ m'vlp s i ng t r ac k of it comp j etely . 
But-(|5mn^ and riot 1 needed anymoreViv i'*-: • 

Such boundaries cannot be easily eliminated, but the 
consequences of making transitions across boundaries can be 
minimised; for example, user B had difficulty restoring the 
state before he closed the browser, while user E had difficulty 
establishing exactly what state the system was in (including 
what had and had not been printed or fully downloaded): 
common ground needs to be restored. 



CONCLUSIONS 

We have considered deterrents to the use of digital libraries, 
and usability issues that are raised by the pragmatics of use in 
a natural setting - taking account of organisational and other 
concerns. One way of considering usability is as the absence of 
any undesirable features of the interaction; therefore, the focus 
in this paper has been mainly on difficulties experienced 
rather than successes, although all users achieved some 
success in their interactions. If some libraries have been 
discussed more than others, it is largely because those 
libraries were used more extensively - a consequence of use 
being discretionary. 

In terms of the Interaction Framework concepts introduced 
earlier, the most important design issues have been found in 
this study to relate to: 

1. Familiarity: users need to be able to rapidly acquire 
understanding of core library features, content and 
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structures, if they are to gain a sense of progress while 
working with the library. 

2. Blind alleys: many interaction sequences did not achieve 
the user’s objectives. This is most obvious when a search 
returns “no matches”, but occurs in some more extended 
interactions. Some interactions that have no material 
outcome achieve improved familiarisation (the user learns 
more about the library structure or contents), but others 
do not. Perhaps more surprisingly, some interactions that 
resulted in a large number of hits failed to achieve the 
user’s objectives because the user was (apparently) 
overwhelmed by choice and retreated from the search 
results page, due to poor discriminability. 

3. Discriminability: forming understandings of the content 
and possibilities in a collection relies on being able to 
discriminate between possibilities. Therefore we need to 
ensure that the potential events are easily discriminated. 

4. Serendipity - finding unexpected interesting results - 
seems to give users a particular sense of making progress. 
Serendipity depends on users . being easily able to 
identify interesting information, which is one aspect of 
discriminability. 

5. Working across boundaries: transition events, where 
one agent (user or computer system) changes context, can 
often be a source of interactional difficulties; measures 
need to be taken to ensure agents maintain common 
ground and understand the consequences of transitions. 

This study has identified many cases where decisions taken by 
computer scientists and librarians have had unanticipated 
consequences. The incremental approach to delivering library 
resources has many positive advantages (e.g. ease of 
introducing new features) but can also result in inconsistency 
as users move between libraries, particularly where that move 
is made so smooth that users are often unaware it has even 
happened. The use of site licenses that permit partial access to 
resources, or access that is temporary, makes systems 
unpredictable to users, so that they cannot develop an 
adequate (usable) understanding of what detailed goals are and 
are not achievable. If theoretical possibilities are to become 
practical solutions, then the kinds of pragmatic issues raised 
by this study need to be taken into account in the design of 
future libraries. 
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